Fixes for Python 3.12, NumPy 2.0, and SciPy 1.14 #1050

guyer · 2024-06-18T17:17:21Z

- Python 3.12 doesn't like SafeConfigParser - Python 2.7 doesn't like versioneer > 0.18

wd15 · 2024-06-18T18:57:37Z

@guyer, you've broken Nix again!

guyer · 2024-06-18T20:09:30Z

@wd15, Nix is broken by design

This library no longer exists in Python 3.12 It looks like it isn't even needed in Python 2.7 to use pickle and redirecting_html.py should go away (but fixed it anyway).

Based on https://github.com/scipy/scipy/pull/20172/files as suggested by https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword

Broken in 2eed418

petsc(4py) 3.21.2 crashes when running tests in parallel: ``` mesh2D (examples.cahnHilliard) Doctest: examples.cahnHilliard.mesh2D ... Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 Internal error 2 in DMUMPS_LOAD_RECV_MSGS 1275068685 52 Internal error 1 in DMUMPS_LOAD_PROCESS_MESSAGE Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 1 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Abort(-99) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -99) - process 0 Internal error 1 in DMUMPS_LOAD_PROCESS_MESSAGE Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 Internal error 2 in DMUMPS_LOAD_RECV_MSGS 1275068685 52 Internal error 1 in DMUMPS_LOAD_PROCESS_MESSAGE Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 Internal error 1 in DMUMPS_LOAD_PROCESS_MESSAGE Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 Internal error 2 in DMUMPS_LOAD_RECV_MSGS 1275068685 52 Internal error 1 in DMUMPS_LOAD_PROCESS_MESSAGE Internal error 1 in DMUMPS_LOAD_RECV_MSGS 1 ```

wd15 · 2024-06-21T19:56:01Z

Nix passes in #1051 with a single tweak. Basically with the master branch with small change to flake.nix. Python still at 3.10.

wd15 · 2024-06-21T20:16:12Z

Nix runs and has some failures (see #1052) when testing this branch with updated Nix to 24.05 with Python 3.12 and Numpy 1.26.4. Latest version of release nixpkgs does not have Numpy 2.0.0, obviously.

wd15 · 2024-06-25T13:40:38Z

@guyer, you're not seeing this test failure anywhere, but on the Nix branch, right?

======================================================================
FAIL: CSR (fipy.matrices.scipyMatrix._ScipyMatrix)
Doctest: fipy.matrices.scipyMatrix._ScipyMatrix.CSR
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/c7ycrgwv039nqglbif98yggx211sdbcl-python3-3.12.3/lib/python3.12/doctest.py", line 2249, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for fipy.matrices.scipyMatrix._ScipyMatrix.CSR
  File "/home/wd15/git/fipy/fipy/matrices/scipyMatrix.py", line 280, in CSR

----------------------------------------------------------------------
File "/home/wd15/git/fipy/fipy/matrices/scipyMatrix.py", line 302, in fipy.matrices.scipyMatrix._ScipyMatrix.CSR
Failed example:
    print(numerix.asarray(cols))
Expected:
    [0 1 2 1 2 0 2]
Got:
    [0 2 1 2 1 2 0]
----------------------------------------------------------------------
File "/home/wd15/git/fipy/fipy/matrices/scipyMatrix.py", line 304, in fipy.matrices.scipyMatrix._ScipyMatrix.CSR
Failed example:
    print(numerix.asarray(data))
Expected:
    [ 12.3         10.           3.           3.14159265   2.96
       2.5          2.2       ]
Got:
    [ 12.3          3.          10.           2.96         3.14159265
       2.2          2.5       ]


----------------------------------------------------------------------

This is with Numpy 1.26.4 and Scipy 1.13.0.

wd15 · 2024-06-25T13:53:00Z

This is with Numpy 1.26.4 and Scipy 1.13.0.

wd15 · 2024-06-25T13:53:19Z

@guyer, can I bring the changes from #1052 into this branch?

guyer · 2024-06-25T14:16:22Z

@guyer, you're not seeing this test failure anywhere, but on the Nix branch, right?

Nope

guyer · 2024-06-25T14:16:38Z

@guyer, can I bring the changes from #1052 into this branch?

@wd15, yes, please

Update Nix build to use Python 3.12, NixOS 24.05. Also, include aarch4-darwin as a possible build. Currently, there is a single test failure in fipy.matrices.scipyMatrix._ScipyMatrix.

Generate the egg_info before executing the tests otherwise setup.py doesn't register the correct version of the test suite, but possibly the older version already available in nixpkgs.

wd15 · 2024-06-25T17:11:51Z

Next step might be to try and build a conda environment with the same numpy and scipy versions as the Nix build to see if it also fails that one test.

guyer · 2024-06-25T19:00:59Z

Next step might be to try and build a conda environment with the same numpy and scipy versions as the Nix build to see if it also fails that one test.

Agreed. I built such an environment on my Mac and get the same error:

python      3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:13:44) [Clang 16.0.6 ]
fipy        3.4.4+41.ge25c210ec
numpy       1.26.4
pysparse    not installed
scipy       1.13.0
matplotlib  3.8.4
mpi4py      not installed
petsc4py    not installed
pyamgx      not installed
PyTrilinos  not installed
mayavi      4.8.2
gmsh        4.12.2
solver      scipy

 :
 :

======================================================================
FAIL: CSR (fipy.matrices.scipyMatrix._ScipyMatrix)
Doctest: fipy.matrices.scipyMatrix._ScipyMatrix.CSR
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/guyer/mambaforge-arm/envs/nix-scipy/lib/python3.12/doctest.py", line 2256, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for fipy.matrices.scipyMatrix._ScipyMatrix.CSR
  File "/Users/guyer/Documents/research/FiPy/fipy/fipy/matrices/scipyMatrix.py", line 280, in CSR

----------------------------------------------------------------------
File "/Users/guyer/Documents/research/FiPy/fipy/fipy/matrices/scipyMatrix.py", line 302, in fipy.matrices.scipyMatrix._ScipyMatrix.CSR
Failed example:
    print(numerix.asarray(cols))
Expected:
    [0 1 2 1 2 0 2]
Got:
    [0 2 1 2 1 2 0]
----------------------------------------------------------------------
File "/Users/guyer/Documents/research/FiPy/fipy/fipy/matrices/scipyMatrix.py", line 304, in fipy.matrices.scipyMatrix._ScipyMatrix.CSR
Failed example:
    print(numerix.asarray(data))
Expected:
    [ 12.3         10.           3.           3.14159265   2.96
       2.5          2.2       ]
Got:
    [ 12.3          3.          10.           2.96         3.14159265
       2.2          2.5       ]


----------------------------------------------------------------------
Ran 541 tests in 78.140s

FAILED (failures=1)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Skipped 38 doctest examples because neither `lsmlib` nor `skfmm` can be found on the $PATH
Skipped 1 doctest examples because `skfmm` must be used to run some tests
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

guyer · 2024-06-25T19:21:37Z

SciPy 1.13.0, in combination with all versions of numpy 1.26.x or numpy 2.0.0, exhibits this issue under Python 3.12 (earlier numpy not an option)

guyer · 2024-06-25T19:28:17Z

The issue goes away with SciPy 1.13.1, with any compatible numpy. Possibly fixed by scipy/scipy#20670.

wd15 · 2024-06-25T19:35:46Z

The issue goes away with SciPy 1.13.1, with any compatible numpy. Possibly fixed by scipy/scipy#20670.

Oh, great. Let me see if I can get a newer scipy into the mix.

guyer added 7 commits June 18, 2024 09:04

Monkeypatch versioneer.py

72b3885

- Python 3.12 doesn't like SafeConfigParser - Python 2.7 doesn't like versioneer > 0.18

Replace obj2sctype

2c7136d

Replace .getsctype() with .dtype

2eed418

Replace NUMERIX.issubclass_() with issubclass()

62bd0ff

Replace itemset() with indexing

4e7923a

Fix obsolete name for infinity

9921bf8

Fix comparison operator representation

55f78f3

guyer requested review from wd15 and tkphd June 18, 2024 17:17

guyer mentioned this pull request Jun 18, 2024

Numpy 2.0.0 breaks things #1049

Closed

guyer added 3 commits June 20, 2024 09:07

Use ins.argrepr for representation of operator variables

0b14ced

Remove future.standard_library

08124c2

This library no longer exists in Python 3.12 It looks like it isn't even needed in Python 2.7 to use pickle and redirecting_html.py should go away (but fixed it anyway).

Update locks

d91ccdc

guyer mentioned this pull request Jun 20, 2024

[WIP] Make compatible with Python 3.12 #1023

Closed

guyer added 11 commits June 20, 2024 11:59

Add return_scalar to __array_wrap__ for NumPy 2.0

29047ee

Add copy= argument to __array__ for NumPy 2.0

69a4e24

Address copy=None for NumPy < 2.0

908825d

Based on https://github.com/scipy/scipy/pull/20172/files as suggested by https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword

Defer return_scalar for OperatorVariables

6e6c68e

Fix polarity of floating CellVariable check

729e2c5

Broken in 2eed418

Replace deprecated numpy.in1d with numpy.isin

130983f

Replace deprecated tol= with rtol= to SciPy Krylov solvers

afd1697

Replace deprecated atol='legacy' for SciPy Krylov solves

43c143a

Remove internal implementation details of numpy.array2string

3a1e316

Override legacy formatting set for tests

0ce2a46

Make directory that already exists

b9f75b8

guyer changed the title ~~issue1049 numpy 2.0.0 breaks things~~ Fixes for Python 3.12, NumPy 2.0, and SciPy 1.14 Jun 21, 2024

guyer added 2 commits June 21, 2024 16:25

Fix case of tarball

4e30696

Fix type of int Variable for Windows

1e411d4

guyer requested review from wd15 and tkphd and removed request for wd15 and tkphd June 21, 2024 23:06

wd15 added 2 commits June 25, 2024 13:00

build(nix): update to python 3.12 and nixos 24.05

21bc31e

Update Nix build to use Python 3.12, NixOS 24.05. Also, include aarch4-darwin as a possible build. Currently, there is a single test failure in fipy.matrices.scipyMatrix._ScipyMatrix.

ci(actions): generate egg_info

e25c210

Generate the egg_info before executing the tests otherwise setup.py doesn't register the correct version of the test suite, but possibly the older version already available in nixpkgs.

wd15 approved these changes Jun 25, 2024

View reviewed changes

guyer merged commit 66774ea into master Jun 25, 2024
20 of 24 checks passed

guyer deleted the issue1049-Numpy_2.0.0_breaks_things branch June 25, 2024 21:58

wd15 mentioned this pull request Jun 26, 2024

Update Scipy version in Nix build to remove test failure #1055

Open

wd15 mentioned this pull request Jul 22, 2024

build(nix): update to using latest Nix master version #1061

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for Python 3.12, NumPy 2.0, and SciPy 1.14 #1050

Fixes for Python 3.12, NumPy 2.0, and SciPy 1.14 #1050

guyer commented Jun 18, 2024 •

edited

Loading

wd15 commented Jun 18, 2024

guyer commented Jun 18, 2024

wd15 commented Jun 21, 2024 •

edited

Loading

wd15 commented Jun 21, 2024

wd15 commented Jun 25, 2024 •

edited

Loading

wd15 commented Jun 25, 2024

wd15 commented Jun 25, 2024

guyer commented Jun 25, 2024

guyer commented Jun 25, 2024

wd15 commented Jun 25, 2024

guyer commented Jun 25, 2024

guyer commented Jun 25, 2024 •

edited

Loading

guyer commented Jun 25, 2024

wd15 commented Jun 25, 2024

Fixes for Python 3.12, NumPy 2.0, and SciPy 1.14 #1050

Fixes for Python 3.12, NumPy 2.0, and SciPy 1.14 #1050

Conversation

guyer commented Jun 18, 2024 • edited Loading

wd15 commented Jun 18, 2024

guyer commented Jun 18, 2024

wd15 commented Jun 21, 2024 • edited Loading

wd15 commented Jun 21, 2024

wd15 commented Jun 25, 2024 • edited Loading

wd15 commented Jun 25, 2024

wd15 commented Jun 25, 2024

guyer commented Jun 25, 2024

guyer commented Jun 25, 2024

wd15 commented Jun 25, 2024

guyer commented Jun 25, 2024

guyer commented Jun 25, 2024 • edited Loading

guyer commented Jun 25, 2024

wd15 commented Jun 25, 2024

guyer commented Jun 18, 2024 •

edited

Loading

wd15 commented Jun 21, 2024 •

edited

Loading

wd15 commented Jun 25, 2024 •

edited

Loading

guyer commented Jun 25, 2024 •

edited

Loading